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Abstract 

Social validity data were collected foi three educational interventions designed 
to increase the academic engaged time of elementary students: a reading tutorial 
intervention, a special education consultation intervention, and a home-based 
consultation intervention. A total of 29 students, 14 parents, 21 teachers, and 5 
pnr -ipals completed questionnaires. All three programs were rated as good 
instructional interventions overall. Aspects often noted as important included tht 
relationships of those involved in the intervention, adequate provision of time for 
discussion, and the screening of participants. Principals had minimal involvement 
in the intervention programs. Estimates of academic improvement due to the 
intervention, and v^illingness to participate in a similar program were varied. 
Potential reasons for these results are discussed. 
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Social Validity Evaluations of Three Interventicns Targeting Increases 

in Academic Engaged Time 

The tenn "social validity" refers to the consumer's reaction to a change or 
intervention; it is the consumer's attitudes or affective outcomes (Wolf, 1978). While 
this variable often is ignored in laboratory*based educational research, it is a very 
important component of applied research, and essential to the generalization of 
research fmdings. Baer (1987) identified social validity as the process that controls 
whether interventions are allowed to begin. For interventions already started, Baer 
viewed social validity as the process allowing the survival of interventions and 
controlling the breadth of adoption achieved by those interventions that do survive. 
The divergence of interests between researchers and practitioners is a serious 
problem, often attributed to researchers striving for statistical rather than practical 
significance (Barlow & Hersen, 1984; Keith, 1988). Increasing the emphasis on social 
validity, particularly by employing a process of social validation when evaluating 
research, would help eliminate this researcher-practitioner gap (Kazdin, 1977). 

Schwartz and Baer (1989) pointed out that soliciting feedback from the 
community during an intervention can lead to a closer relationship between 
participants and interveners. If the feedback is utilized, it can provide the audience 
with active, shared control of the intervention program. Schwartz and Baer noted an 
encouraging increase in the use of social validation as an aspect of research studies, 
particularly in the area of behavior analysis. They further noted the need for social 
validity assessments to expand the range of consumers assessed and the development 
of more homogeneous and objective techniques for assessing social validity. 

Kazdin (1977) outlined three important facets that need to be aasessed when 
deteimining the social validity of interventions. The first area is the focus of the 
intervention, specifically whether the target behaviors are important to those in the 
natural environment. Second, the acceptability of the procedures used in the 
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intervention needs to be determined, both for those implementing and for those 
receiving the intervention. Last, the importance of the behavior change needs to be 
determined by those in daily contact with the subject or by comparing the subject's 
performance change with that of peers. Wolf (1978) outlined the need to socially 
validate procedures at three similar levels: goals, procedures, and effects. 

During the 1987-88 academic year, the Instructional Alternatives Project 
implemented three interventions with elementary students who were receiving 
special education services. The three interventions were a reading tutorial 
interveution (Weiss, Thurlow. Christenson. & Ysseldyke, 1988), a special education 
\ consultation intervention (Ysseldyke. Christenson, Shrincr, & Gorncy, 1989), and a 

home-based consultation intervention (Christenson, Thurlow, Cleary, & Ysseldyke, 
All interventions were assessed in terms of their effects on achievement and 
academic engaged time. 

The reading intervention was designed to increase the academic engaged time 
of nine elementary students with reading difficulties through sessions with 
volunteer tutors. At least 36 20-minute tutoring sessions were held over an 11-week 
period, during which tutors used paired reading, supported by flashcarus, and a 
variety of other methods. For a detailed description of this intervention and the 
results obtained, see Weiss et al. (1988). 

The special education consultation intervention >vas an individualized training 
and collaborative problem-solving process involving seven special education 
teachers, each with one or two intervention students and one control student. 
Implementation was done iri three stages from November to April: (1) curriculum- 
based measurement (CBM) only, (2) CBM plus paper feedback about the student's 
academic engaged time, and (3) instructional consultation. The main goal of the 
special education consultation intervention was to inc.ease students' academic 
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engaged time through individual instructional strategies. A thorough description of 
this intervention can be found in Ysseldyke et al. (1989). 

The home-based consultation intervention was conducted to determine the 
effectiveness of a structured homework intervention in increasing academic 
responding in nine mildly handicapped students. A focused effort also was made to 
document home and school factors that contributed to these students' noncompletion 
of assigned work. This 12-week study primarily involved consultants working 
individually with the parents of the students to de.sign interventions aimed at 
increasing homework completion and reading time at home. Christenson et al. (1989) 
provide a thorough description of this intervention and its results. 

For each of the three interventions, information was obtained from students, 
parents, and teachers to assess social validity. Subjective evaluations were used to 
assess different facets of the interventions that needed social validation. Results are 
presented in a descriptive format. 

Method 

Subjects 

Subjects for this study were students involved in each intervention, the 
students' parents (with the exception of the special education consultation 
intervention), the teachers of the students, and the principals in the schools in 
which students were located. The numbers of possible subjects for each intervention 
are shown in Table 1. Also shown in this table are the actual numbers of respondents 
from whom data were obtained. 

All students were in grades 2-6 and receiving special education services when 
the study was conducted. Teachers were either in special education resource rooms 
(n - 8) or general education classrooms (n = 13; one mainstream teacher participated 
in both the tutoring and home-based consultation interventions). Teachers 
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completed a social validity questionnaire for each student in an intervention. Thus, 
for the special education instructional consultation intervention, a teacher who had 
two subjects in the intervention completed a questionnaire for each student. The 12 
appearing in Table 1 for the number of teachers in the special education 
consultation intervention reflects 12 teacher responses (possible and returned) for 
the 7 teachers in the study. All students and teachers were from six schools in a 
suburban school district. Principals from all of these schools were sent social 
validity questionnaires. However, since not all interventions were conducted in all 
schools, the number of possible principals varied with the intervention. 

Students for the interventions were recommended by teachers based upon who 
they felt could benefit from the intervention. All teachers were volunteers, except 
for one in the special education consultation intervention; this teacher was selected 
by the district'^ special education director to ensure that each school building had at 
least one special education teacher in that intervention. The project coordinators 
met with the teachers before the interventions began, and maintained regular 
phone contact with them throughout the study. A bnef summary outline of the 
interventions as presented before implementation is presented in Appendix A. 
Instruments and Procedures 

All social validity ratings were made at the end of the school year, after all 
interventions had been completed. For students, questions were read by data 
collectors working in the schools. Parents weu asked the social validity questions by 
the same data collectors over the telephone; \1 the parents declined to respond, no 
further contacts were made. For both principals and teachers, questionnaiies were 
used. One attempt was made to follow-up on nonresponses. 

Social validity questionnaires for the reading intervention were completed by 
students, parents, and reading teachers. The student questionnaire asked for ratings 
to eight statements having to do with enjoying the tutoring and increased reading 

s 




Table 1 

Potential and Actual Number of Respnnsses to Questionnaires 



Type of 
Respondent 



Reading 



Special Education 
Consultation 



Home-Based 
Consultation 



Possible Responded Possible Responded Possible Responded 



Student 
Parent 
Teacher 
Principal 



9 
9 
9 
4 



8 
8 
9 
3 



12 

X 

12 
6 



12 

X 

12 
5 



9 
9 
9 
4 



9 
6 
9 
3 



ability as a result of the tutoring (see Appendix B). Parents and teachers received 
similar questionnaires that required six ratings related to opinions about volunteer 
tutoring as an intervention in general, and five ratings on the benefits of the 
intervention for this student. Comments and suggestions related to the tutorial 
program also were requested (see Appendix B). 

Questionnaires investigating the social validity of the special education 
consultation intervention were completed ^ by the intervention students and their 
teachers. There were eight items on the student questionnaire. The statements 
addressed disruptions in the classroom due to the observations and students enjoying 
having visitors in cl?ss and doing the CBM activities (i.e.. timings and graphs). 
Teachers filled out questionnaires for each intervention student. These forms asked 
for eight ratings on teacher perceptions of the usefulness of each of the three stages 
of the intervention, and whether they would participate in similar program. There 
were also three questions asking for comments and/or suggestions for improving the 
intervisntion (see Appendix C). 

Students, parents, and the students' mainstream teachers each completed a 
social validity questionnaire for the home-based consultation intervention. The 
students' questionnaires had eight items asking for ratings of how their structuring 
of homework assignments, and attitude toward homework had changed since the 
beginning of the project (see Appendix D). They also were asked whether they would 
like to participate in a project like this again. Parents and teacheis were given the 
same questionnaire. It asked for ratings on nine items that paralleled the student 
items: How has the students' attitude toward, and completion of. homework changed 
since the beginning of the project, and would you participate again? They also were 
asked about the helpfulness of the consultant and for comments and suggestions for 
improving the program. 
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Principals of the schools involved in the interventions also were mailed a 
bri^f social validity questionnaire (see Appendix E). They were asked for ratings of 
the amount of feedback they had received from those involved in the project, to what 
extent they had been involved in the project, and to what extent they thought the 
student had benefited from the project. They also were asked to provide their 
opinions about the most positive aspect of the program, suggestions for improvement 
or changes, and general comments. The principal questionnaires were identical for 
all three interventions. 

Results 

Reading Tutorial Intervention 

Student responses to the social validity questionnaires are summarized in Table 
2. In general, the responses indicated that students enjoyed the program and 
thought that it had improved their reading. Responses related to a desire for future 
participation and perceived desire of friends to participate were equivocal. 
Responses about reading a greater variety of books tended toward agreement. 

Responses to the parent questionnaire are summarized in Table 3. Parent 
responses to volunteer tutoring as an intervention were more mixed. Generally 
parents indicated the students were reading new words better, understanding more, 
and having a positive atcitude, but not necessarily reading more books or a greater 
variety of books. There was a trend toward indicating that reading occurred more 
frequently. Parents indicated in writte^i comments that six of the eight children had 
told them that they liked the tutor or tutoring. Other things the parents reported 
hearing were that their children liked getting out of class, and that they are now 
reading a greater variety jf books. One parent reported hearing very little about the 
program. 
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Table 2 

Student Responses to Reading Interventi on Social Validity Qiiesiionnaire' 



Ratine 

Strongly Strongly 
Item Disagree Disagree Agree Agree 



Enjoyed woiking with tutor. — 


1 (12.5) 


5 (62.5) 


2 (25.0) 


Tutoring has helped reading. 




4 (50.0) 


4 (50.0) 


Like leaving class. — 


2 (25.0) 


4 (50.0) 


2 (25.0) 


Friends want tutoring. 1 (12.5) 


4 (50.0) 


3 (37.5) 


* « * 


Want to have tutor again. — 


3 (37.5) 


2 (25.0) 


3 (37.5) 


Improved understanding when reading. 




6 (75.0) 


2 (25.0) 


Improved reading of new words. — 


2 (25.0) 


4 (50.0) 


2 (25.0) 


Greater variety of books read. 


3 (37.5) 


4 (50.0) 


1 (12.5) 



^Entries are numbers and perceiilages of students giving a particular response (n=8). 



Table 3 

Parent Rfcsponses to Reading Intervention Social Validity Questionnaire 



EMlm£ 

Strongly Strongly 
Item Disagree Disagree Agree Agree 



Volunteer tutors have adequate skills. 


- - - 


7 (87.5) 


1 (12.5) 


Quantity of tutoring was sufficient. 


2 (25.0) 


4 (50.0) 


2 (25.0) 


Tutoring offsets missed class time. 


2 (25.0) 


4 (50.0) 


2 (25.0) 


Oral reading practice helps academics. 




6 (75.0) 


2 (25.0) 


Volunteers provide dependable help. 




7 (87.5) 


1 (12.5) 


Willingness to participate. 




4 (50.0) 


4 (50.0) 


Improved understanding when reading. --- 


1 (25.0) 


5 (62.5) 


2 (25.0) 


Improved reading of new words. 




3 (37.5) 


5 (62.5) 


Greater variety of books read. — 


5 (62.5) 




3 (37.5) 


Reading is more frequent. 


3 (37.5) 


4 (50.0) 


1 (17.5) 


Improved altitude toward reading. 


1 (12.5) 


5 (62.5) 


2 (25.0) 



^Entries are numbers and percentages of parents giving a particular response (n=8). 
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Five parents reported hearing nothing from their child about aspects of the 
program that could be improved. Those suggesting changes mentioned: (1) having 
the home involved to improve student confidence, and (?) making it easier for the 
student to leave class. 

Table 4 is a summary of teacher questionnaire results. Volunteer tutoring was 
regarded positively as an intervention. Responses about the results of the tutoring 
program indicated that tutors were reliable and adequately skilled. The quantity and 
method of tutoring was seen as good enough to offset missed class time, and improve 
student attitudes, reading of new words, and academics in general. When asked what 
they considered the most positive aspects of the program, all five teachers who 
responded mentioned the positive relationship established between the student and 
the tutor and/or the benefits of one-to-one academic work. Seven of the nine 
teachers were willing to participate again in a similar intervention. 

Only two teachers offered suggestions for improvement or change. One 
suggestion was to provide more information about the program before it begins. 
Another was that there should be more communication between the tutor and 
teacher during the intervention. It was noted that one student missed language 
instruction while out of the room for tutoring. Three additional comments were 
made. One teacher resented the frequency of the observations and the lack of 
feedback, and thought the school should be compensated for taking part in the 
project. The other two comments were positive, mentioning the program being 
considerate of teacher schedules, aud that the tutor-student relationship was 
particularly valuable because of the present home situation of the student. 
Special Education Consultation Intervention 

A summary of the students' responses to the social validity questionnaire for 
the special education consultation intervention is provided in Table 5. Overall, 
responses indicated that students did not consider visitors in class disruptive and that 
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Tabic 4 

Tp.ap.hp.r Response58 to Reading Intervention Social Validity Questionnaire 



Rating 

Strongly Strongly 
Item Disagree Disagree Agree Agree 



Volunteer tutors bat/e adequate skills'' 






5 (83.3) 


I 


(16.7) 


Quantity of tutoring was sufficient.^ 




1 (14.3) 


5 (71.4) 


1 


(14.3) 


Tutoring offsets missed class time. 




2 (22.2) 


5 (55.6) 


2 


(22.2) 


Oral reading practice helps academics. 






8 (88.9) 


1 


(11.1) 


Volunteers provide dependable help.^ 




1 (12.5) 


6 (75.0) 


1 


(12.5) 


Willingness to participate again. 




2 (22.2) 


5 (55.5) 


2 


(22.2) 


Improved understanding of reading.^ 




2 (28.6) 


5 (71.4) 






Improved reading of new words. ^ 




2 (25.0) 


5 (62.5) 


1 


(12.5) 


Greater variety of books read.^ 


2 (25.0) 


3 (37.5) 


2 (25.0) 


1 


(12.5) 


Reading is more frequent. 


1 (11.1) 


4 (44.4) 


4 (44.4) 






Improved attitude toward reading.^ 


1 (12.5) 




5 (62.5) 


2 


(25.0) 



^Entries are numbers and percentages of teachers giving a particular response 
(n=9). 

^Nonresponses occurred for these items, ranging from 1 to 3 nonr^sponses. 
Percentages are based on the number responding. 
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Table 5 

Student Responses to Special Education Consultation Interventitin Social Validity 
Ouestionnairea 



Rating 

Strongly Strongly 
Item Disagree Disagree Agree Agree 



Observers hinder attention. 


1 ( 8) 


7 (58) 


3 (25) 


1 ( 


8) 


Schoolwork is altered during 
observations. 


3 (25) 


9 (75) 








Student treated differently 
during observations. 


2 (17) 


9 (75) 


1 ( 8) 






Student enjoyed observations. 




7 (58) 


5 (42) 






Computer was not disruptive. 




1 ( 8) 


10 (83) 


1 ( 


8) 


Timings were enjoyed. 




2 (17) 


9 (75) 


1 ( 


8) 


Graphing was enjoyed. 




1 ( 8) 


9 (75) 


2 (17) 


Want for future timings. 


2 (17) 


4 (33 : 


5 (42) 


1 ( 


8) 



^Entrifs are numbers and percentages of students giving a particular response 
(n=12). 
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they liked the activities associated with cur;iculum-based measurement (e.g., 
timings, graphing). The items asking about continuing participation in these kinds 
of activities and enjoying classroom visitors received responses split almost equally 
between positive and negative. 

Table 6 is a summary of teacher responses to the questionnaire. Results 
indicate support for curriculum-based measurement as an instructional aid or 
strategy. A collaborative model was also seen as the preferred mode of consultation. 
Equivocal responses were obtained when teachers were asked about the instructional 
relevance of computer print-outs of observed behavior, and when asked about 
participation in similar consultation programs. 

All seven teachers responding to the question asking about positive aspects of 
the program mentioned the time spent meeting with the consultants and the mutual 
sharing of ideas and interventions. Teachers indicated that this served as an impetus 
to take time to think about that particular student and to try new interventions. Also 
mentioned as positive was the focus on time on task. One teacher indicated that 
observational summaries were a positive aspect of the intervention. 

Many suggestions were given by the six teachers responding to the question 
asking about facets of the program that could be improved or changed. Time for 
consultation was a major concern, mentioned on five forms. Suggestions included 
building consultation time into teachers' schedules, involving the regular education 
teacher in the consultation time, increasing the length of time between 
consultations in order for intervention results to be seen more clearly, and 
providing a clearer explanation of the time requirements of the program before 
implementation. Other suggestions were to do timings on the subject material used in 
the special education classroom (not necessarily reading) and to allow the 
consultation model to evolve into cither an "expert consultation" or "collaborative 
consultant" framework, rather than designing it. 
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Table 6 

Teacher Respnnsfts to Special r ii u-atinn Cnnsnltatinn Intervention Social Validity 
QiiRstinnnaire a 



Hating 

Strongly Strongly 
Itcm*^ Disagree Disagree Agree Agree 



CBM helps monitor progress. 




9(90.0) 


1(10.0) 


r^P\>f limine Hpt^rminp ^Ifill levelK 


U 9.1) 


10(90.9) 




CBM helps make instructional 
decisions. 


1( 9.1) 


9(81.8) 


K 9.1) 


— Mite _ . A — A A « a A M 1 ■% 

Computer printouts neip 
instructional planning. 


5(50.0) 


4(40.0) 


1(10.0) 


Consultants help develop 
interventions 


6(06.7) 


3(33.3) 




Collaborative consultation is more 
effective than an expert model. 


1(10.0) 


5(50.0) 


4(40.0) 


CBM graphs motivate students. 




7(63.6) 


4(36.4) 


Willingness to participate again. 


3(33.3) 2(22.2) 


2(22.2) 


2(22.2) 


^Entries arc numbers and percentages 
(n=12). 

hNonresponses occurred for every item 


of teachers giving a 
with a maximum of 3 


particular response 
nonresponses on 1 item. 
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Five questionnaires included additional teacher comments. In general, the 
comments reflected discomfort with the number of observations and length of the 
program. One teacher commented about bein^ unaware that the program would last 
quite so long and involve so many people, observations, questionnaires. Other 
responses raised questions about the extent to which the data accurately reflected 
student progress; teachers expressed feeling a pressure to show big improvements in 
the data: *'I felt I had to be much more effective and student had to improve 
commensurate with all the extra help and he didn't improve significantly." 
Home-Based Consultation Intervention 

Student responses to the social validity questionnaire for the homework 
intervention are summarized in Table 7. Ratings indicated minimal change overall 
in attitude toward homework and school work since the beginning of the 
intervention. Although there was a trend toward homework being given more 
regularly, an increased willingness to do homework, better organized homework, and 
parents helping more, equivocal responses were given to questions about 
independent completion of homework, schoolwork being more enjoyable, and the 
students* willingness to participate in a similar project. Student responses did 
indicate that more assignments were done each day and that more of the homework 
was completed correctly since the beginning of the project. 

Parent responses (see Table 8) were similar to the student responses. Ratings 
of their children's attitudes toward and organization of homework varied, as did their 
ratings of the usefulness of a homework consultant. Most parents did indicate that 
their children completed a greater amount of their homework and thai the 
consultants helped in planning homework. They also indicated that they would 
participate in another homework project. 
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Table 7 

Student Responses to Hnme-Rased Consul tation Social Validity Questionnaire 



Rating 

Strongly Strongly 
Item Disagree Disagree Agree Agree 



Homework is given more regularly. 








^^^^ i ^ 


Increased willingness to do homework. 2(22.2) 


1(11.1) 


5(55.6) 


1(11.1) 


Homework done without help. 


1(11.1) 


3(33.3) 


3(33.3) 


2(22.2) 


Homework is better organized. 


1(11.1) 


2(22.2) 


5(55.6) 


1(11.1) 


More ol' homework is correct. 


1(11.1) 


1(11.1) 


. 4(44.4) 


3(33.3) 


Increased completion of homework. 


1(11.1) 




5(55.6) 


3(33.3) 


Parent(3) help more with 
organization. 


2(22.2) 


1(11.1) 


2(22.:'.) 


4(44.4) 


Schoolwork is more enjoyable. 


2(22.2) 


2(22.2) 


3(33.3) 


2(22.2) 


Willingness to participate again. 


2(22.2) 


3(33.3) 


3(33.3) 


1(11.1) 



^Entries are numbers and percentages of studen s giving a particular response (ii=9). 
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Table 8 

Parent Responses to Home-Based Cnnsuhation Social Validity Ouestinnnairea 



'item 


Strongly 
Disagree 


Disagree 


Agree 


Strongly 
Agree 


Homework is given more regularly. 


1(16.7) 


1(16.7) 


3(50.0) 


1(16.7) 


Increased compliance in starting 
homework. 


1(16.7) 


2(33.3) 


1(16.7) 


2(33.3) 


More independent study habits. 




2(33.3) 


3(50.0) 


1(16.7) 


Materials are better organized. 




2(33.3) 


3(50.0) 


1(16.7) 


Greater amount of homework 
completion. 




1(16.7) 


4(66.7) 


1(16.7) 


Homework is more accurate. 




2(33.3) 


3(50.0) 


1(16.7) 


Improved attitude toward homework. 


1(16.7) 


2(33.3) 


2(33.3) 


1(16.7) 


Improved attitude toward school. 




1(16.7) 


4(66.7) 


1(16.7) 


Consultant helped plan homework.^ 




1(20.0) 


2(40.0) 


2(20.0) 


Willingness to participate again. 




1(16.7) 


3(50.0) 


2(33.3) 


^Entries are numbers and percentages of students 
^ There was 1 nonresponse for this item. 


giving a particular response (n=:6). 
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All five responding parents commented on the procedures suggested by the 
consultants (e.g., structured environment for homework or home-school monitoring 
of iiomework) or the relationship between the student and the consultant as the most 
positive aspect of the program. Other positives included improved student 
confidence, improved work completion, and improved communication with the 
teacher. 

Four suggestions were made for improvement in the program. They were: 
better screening, more communication between the home and the school, decreasing 
the length of the program, and empha'?izing helping children to have a better 
attitude about homework. 

Several comments were offered by the home-based consultation intervention 
parents. One parent thought the child was ill-suited for the program. Another 
offered the suggestion: "How about a class in study habits?" Three of the five 
responses were positive and indicated their appreciation for the consultant. 

The teachers' responses to the social validity questionnaire for the homework 
intervention showed increased compliance in starting homework (see Table 9). 
There was a general disagreement with the statement that student's materials are 
better organized, or that there has been an increase in independent study habits. 
There was a consistent trend toward seeing greater homework completion, more 
accurate homework, and imp'oved attitudes. A positive trend also was evident when 
teachers were asked about the usefulness of a consultant. Most teachers indicated a 
willingness to participate in a similar intervention in the future. 

Teachers viewed the increased involvement of the parents in homework 
completion and the increased communication between the home aid the school as 
the most positive aspects of the program (4 of 6 responses). Getting the parents to 
appreciate the need for homewoik and the consultants' involvement with the 
parents and child also were seen as positive. 
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Table 9 

Teacher Respnn?ses to Hnme-Rased Consultation Snr ial Validity Ouestionnaire a 



Ratipg 



Item 


Strongly 
Disagree 


Disagree 


Agree 


Strongl; 
Agree 


1. 


Since beginning of project: 












a. Homework is given more 
resularlv,^ 




3(37.5) 


3(37.5) 


2(25.0) 




b. Increased compliance in 

oicllillij^ iiUiilw Mr Ul Jit 


1(16.7) 




5(83.3) 


m m » 




c. More independent study 
habits.^ 


1(12.5) 


5(62.5) 


2(25.0) 


mm • 




d. Materials are better organized. 


1(11.1) 


6(66.7) 


2(22.2) 


m m m 




e. Greater amount of homework 
completion. 


1(11.1) 


2(22.2) 


4(44.4) 


2(22.2) 




f. Homework is more accurate. 


1(11.1) 


2(22.2) 


5(55.6) 


1(11.1) 




g. Improved attitude toward 
homework. 


1(11.1) 


2(22.2) 


5(55.6) 


1(11.1) 




h. Improved attitude toward 
school. 


1(11.1) 


2(22.2) 


6(66.7) 




2. 


Consultant helped plan homework. 




3(33.3) 


5(55.6) 


1(11.1) 


o 

^ • 


Willingness to participate again.^ 


2(28.6) 




4(57.1) 


1(14.3) 



^Entries are numbers and percentages of students giving a particular response (n=9). 
^Nonresponses occurred for these items, ranging from 1 to 3 nonrcsponses. 
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Only two teachers offered suggestions for improving the program. These 
suggestions were to provide for mere communication and support for the parents of 
students who need extra h^^lp and to do more individualized evaluation of and 
planning for student needs. 

Additional comments were offered by only two teachers; they addressed 
specific student needs. One comment mentioned th"* the student is receiving 
homework because her patents requested no modification in the curriculum and this 
has made work increasingly difficult for the student. The other comment was that 
the student would not comply with the requirements of program. 
Principal's Responses 

A limited number of responses was obtained from the principals (see Table 10). 
Three principals responded to the reading intervention questionnaire. The ratings 
showed no involvement in the project, limited feedback from those involved and 
mixed feelings about benefit to the student. No responses were given to the item 
asking about positive aspects of the program. Only one response was given to the 
item asking for suggestions for improvement; this response suggested more 
communication between the project directors and the volunteers, and leys reliance 
upon school personnel for organization. The only comment was one principal 
indicating a preference to not participate again. 

The five principals' responses related to the special education consultation 
intervention showed limited involvement, some feedback, and generally positive 
feelings about the program's benefit to students. The most positive aspects of the 
program were repc *edly the children's enjoynicnt of consultants' attention, the 
opportunity to implement the creative ideas, and suggestions received in order to 
help the student, and reports of improvement by parent(s). Suggestions for change 
included adding regular updates to principals about the program and the use of more 
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Table 10 

Princinals' Responses to Social Validity Ou estionnaires a 



Rating 



Item Very 

Not at All Sometimes Frequently Frequently 



Extent of feedback you recei' cd: 

Reading 1 (33.3) 2 (66.7) 

Special Education 1 (20.0) 4 (8G.0) 
Consultation 

Home-Based 2 (66.7) 1 (33.3) 
Consultation 

Extent of your involvement: 

Reading 3 (100.0) 

Special Education 2 (40.0) 3 (60.0) 
Consultation 

Home-Based 2 (66.7) 1 (33.3) 
Consultation 

Extent of benefit to student(s):^ 

Reading --- 1 (50.0) 1 (50.0) 

Special Education --- 1 (25.0) 3 (75.0) 

Consultation 

Home-Based --- --- 2 (100.0) 

Consultation 



^For the Reading and Home-Based Consultation interventions, n = 3; for the Special 
Education Consultation Intervention, n - 5. 

^One no response occurred to this item for each intervention. 
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experienced consultants. A variety of comments were received, varying from "I 
would not participate in a similar project" to "From what I've learned the information 
has been helpful and useful." 

Three principals responded to the home-based consultation social validity 
questionnaire. Their ratings indicated little involvement in or feedback from the 
program, but two of them thought the project frequently benefited the student. 
Principals mentioned home visits by consultants, extra help, and increased parental 
support of homework as the most positive aspects of the program. There were no 
suggestions for improvement and there was only one comment: "From what I've 
heard, it has iu fact been helpful." 

Discussion 

Social validity research is an important yet difficult aspect of applied 
rtsearch. Conducting research in applied settings typically is fraught with its own 
difficulties, many of them logistical in nature. Even with volunteers as participants, 
as time passes the implementation of an intervention often becomes trying, with 
many of the complexities introduced by any change process occurring (see Hall & 
Hord, 1987; Rossman, Corbett, & Firestone, 1988). On top of all this, researchers add 
the evaluation process, which requires that participants do a little more. When these 
events occur over the span of a school year, and are completed in the spring when 
everything else is occurring 'n the schools, social validity data are difficult to get 
and may be less positive than normally expected. Considerable efforts were made to 
obtain responses from all participants, yet some individuals remained 
nonrcspondents. In at least one case, the nonrespondent refused to respond because 
of the belief that the questions surely had already been answered as part of 
conversations with consultants. The findings must be viewed in light of these 
constraints. 
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Opinions about the social validity of the reading intervention were generally 
similar when student, parent and teacher responses were compared. The /olunteer 
tutor program was viewed as a positive, useful, and enjoyable intervention as a 
whole, with noticeably positive academic results. An unexpected finding was that the 
relationship between the tutor and the tutee was viewed as the most positive aspect of 
the program. This relationship may be the reason for the students' more positive 
attitudes toward reading that was noted by the teachers and parents. 

Th^ special education consultation intervention was viewed as a worthwhile 
and effective intervention overall. Curriculum*based measurement procedures and 
collaborative consultation were viewed positively as ways to determine and monitor 
skill levels and for designing interventions. Meeting with consultants to share ideas, 
particularly ideas about intervenJons, was the most positive aspect of the program, 
but providing for the time to meet was the biggest problem encountered. There was a 
wide range of opinions about the intrusiveness of the observational systems 
employed. The students and most teachers did not view i^ as a problem, but some 
teachers resented and felt pressured by it. While the observational methods used in 
this study would not necessarily be required for every intervention implementation, 
a clear description of the time requirements and number of needed classroom 
observations is an important element of the pre-intervention explanations. The fact 
that this was done by the project coordinator indicates that beyond this, it may be 
necessary to repeat the description throughout the intervention, and then just 
expect ihat some complaints still will be made. Despite the overall positive nature of 
the ratings, the teachers did not positively rate their willingness to participate in a 
similar program, perhaps due to the time requirements i ivolved. 

Responses to the social validity questionnaires for the home-based 
consultation intervention varied considerably. Neither the students nor the parents 
noted an improvement in organization, but both seemed to recognize improved 
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homework completion. The teachers responses also were varied. The parents 
considered the most important aspects of the intervention to be the relationship 
between the student and consultant* and the consultants' ideas, while the teachers 
saw the involvement of the parents in the homework process and the parents' 
communication with the school as the most positive aspects of the program. Further 
investigation is needed to determine whether the level of involvement in the 
program, opinions already held about the source of school problems, or some other 
factors account for these differing views. 

While enlisting the support of the principals in the schools where 
interventions were undertaken was very important, when surveyed at the end of the 
year, the principals indicated that they had very little involvement in or feedback 
about the interventions after they were in place. There was little consensus in the 
perceptions of social validity expressed by those principals who did respond. 
Clarifying what project information will be conveyed to the principal and who is 
responsible for this could be an important factor in enhancing the principal's view 
of the social validity of thw intervention programs. 

Social validity information across the three interventions is characterized by 
some definite similarities. All three programs were viewed overall as using good 
instructional interventions - they were seen to be "good ideas " On the other hand, 
these programs are not always rated as useful given the situation of the student, 
parent or teacher involved. The limited number of people involved in these 
programs makes assurance of full cooperation by those involved essential for 
consistent results. Another factor that consistently appeared to be important, and in 
these studies frequently was referred to as the most positive feature of the 
interventions, was the relationship between the person implementing the 
intervention and the given student, parent or teacher. The "human factor" arises 
again with the selection, preparation and matching of all parties involved being a 
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very important factor in social validity. This factor involving the forming of 
relationships was consistently important across the method of intervention and the 
target person for the intervention (tutoring * student, special education consultation 
* teacher, home*based consultation - parent). 

Perhaps one of the most telling ratings of social validity is whether people are 
willing to participate in another similar project. Both parent questionnaires 
(reading and home-based consultation) indicated a willingness to do so. Teachers 
expressed a aesire to participate again only in the tutoring program, the 
intervention least demanding of their time. Students' responses on these items were 
mixed for all of the programs, perhaps indicating a negative response to being 
singled out for services. 

There are several me;hodological obstacles in social validity research. The 
questionnaire format used in this study relied on subjective evaluation, and thus was 
susceptible to the raters* biases. Individual items also may have been too global, or 
not as directly related to criteria as is optimal. It may have been beneficial in these 
interventions to combine social validity measures as part of an ongoing evaluation. 
Midpoint, or monthly questionnaires would have provided comparison data, and 
opportunity for making valuable program modifications. Despite these difficulties, 
determining the clinical and social significance of our interventions is an essential 
component in developing interventions that will be used for the benefit of students. 
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Instructional Alternatives Project 
Intervention Proposal 
1987-1988 

The goal of all interventions is to increase handicapped students' 
academic engaged time* Each intervention takes a different approach* 



Reading 

Goal: 

Persons Involved: 



To provide additional leading practice for special 
education students 

Special education students in grades 1 - 6> who in their 
teacher's opinion, could show improvement if given 
more practice opportunities in reading. 

Volunteer Tutors for ^ days per week, 30 minutes daily 
per student 



Timeline: October • April 
Curriculum : &M£d Measurement/interven tion Planning 



Goal: 

teaching strategies. 
Persons involved: 



To provide special education teachers frequent 
information on student's performance and an 
opportunity to assess effectiveness of different 



Mildly handicapped students in grades 1 - 6. 

Special education teachers who receive training in CBM 
implementation and monitoring 



Timeline. October 
,^9mgWPrk Intervention 



April 



Goal: 



Persons involved: 



To assist parcrts/guardians in structuring completion 
of schoolwork 

Special education students in grades 1 - 6. whoise 
parents arc interested participants and whose teachers 
believe completion of work and practice at home will 
assist (make an academic achievement difference for) 
the student 



Timeline: 



16 weeks — approximately once per week 
meetings with research personnel 



parent 



Appendix B 



Social Validity Questionnaires for the Reading (Tutorial) Intervention 



Reading - S: Student Questionnaire 
Reading - P: Parent Questionnaire 
Reading - T: Teacher Questionnaire 
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student Name 

Reading - S 

We want to know what you liked about your tutoring in reading. Please circle 
the number on the right that best describes your thoughts for each sentence. 



Note; SD = Strongly Disagree; D = Disagree; A = Agree; SA = Strongly Agree 



1. I enjoyed working with my tutor(s). 

2. Working with my tutor(s) has helped me become a 
better reader. 

3. I liked leaving the class to be tutored. 

4. My friends think having a tutor would be fun. 

5. I would like to have a tutor again. 



SD 
1 

1 
1 
1 
1 



2 

2 
2 
2 
2 



3 

3 
3 
3 
3 



SA 



6. Since my tutor started working with me ; 

a. I understand what I read better than before. 

b. I can read new words better than before. 

c. I read more different kinds of books now. 



1 
1 
1 



2 
2 
2 



3 4 
3 4 
3 4 
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17/JW.4 



student Name 



Reading - P 

We are interested in your perceptions of the reading tutorial program. Please 
circle the number on the right that best describes your agreement with c^ch 
statement on the left. 



Note: SD * Strongly Disagree; D « Disagree; A = Agree; SA = Strongly Agret 



1. Volunteer tutors have adequate skills to — ~ 
provide supplemental instruction. 12 3 4 

2. Providing an additional 20-30 minutes of reading 
instruction daily for 12 weeks is sufficient to 

produce change. 12 3 4 

3. Pulling the child out of class' is more than 
offset by having the child get individual 

instruction. 12 3 4 

4. Children profit academically from the 
opportunity to engage in additional oral 

reading practice. 1 2 3.4 

5. Volunteers are a dependable source of 

supplemental classroom help. 12 3 4 

6. I would have my child participate again 

In a reading tutorial program. 12 3 4 

7. Since the tutor program began ; 

a. The child understands more of what he or she 

reads. 12 3 4 

b. The child reads new words better than 

before. 12 3 4 

c. The child reads a greater variety of books 

now. 12 3 4 

d. The child reads more often than before. 12 3 4 

e. The child's attitude toward reading is more 

positive. 12 3 4 

8. What kinds of positive things did your child tell you about the tutorial? 



9. What kinds of things did your child tell you about the tutorial that might 
be improved or changed. 



10. Any other comments? 

17/JW.3.1 
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student Name 



Reading - T 

We are interested in your perceptions of the reading tutorial program. 
Please circle the number on the right that best describes your agreement with 
each statement on the left. 



Note: SD = Strongly Disagree; D « Disagree; A = Agree; SA = Strongly Agree 



1. Volunteer tutors have adequate skills to ~ -A_ M 
provide supplemental instruction. 12 3 4 

2. Providing an additional 20-30 minutes of reading 
instruction daily for 12 weeks is sufficient to 

produce change. 12 3 4 

3. It is a positive trade-off to pull a student 

out of class to provide indivioual instruction. 1^34 

4. Students profit academically from the 
opportunity to engage in additional oral 

reading practice. 12 3 4 

5. Volunteers are a dependable source of 

supplemental classroom help. 12 3 4 

6. I would have my student participate again 

in a reading tutorial program. 12 3 4 

7. Since the tutor program began ; 

a. The student understands more of what he or she 

reads. 12 3 4 

b. The student reads new words better than 

before. 12 3 4 

c. The student reads a greater variety of books 

now. 12 3 4 

d. The student reads more often than before. 12 3 4 

e. The student's attitude toward reading is more 

positive. 12 3 4 

8. What have been the most positive aspects of the tutorial program? 



9. What facets of the program should be improved or changed? 



10. Any other comments? 
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Social Validity Questionnaires for the Special Education 
Consultation Intervention 



Consultation - S; Student Questionnaire 
Consultation - T: Teacher Questionnaire 



student Name 



Consultation - S 



We want to know what you liked about some changes In your special class. Please 
circle the number that best describes your thoughts. 



Note: SD * Strongly Disagree; D « Disagree; A « Agree; SA * Strongly Agree 



1. It is hard to pay attention wheti visitors come 
into the classroom. 

2. My teacher gives me different kinds of work 
when visitors come Into the classroom. 

3. My teacher treats me differently when visitors 
come into the classroom. 

4. I like having visitors in the classroom. 



5. The computer used by people in my room did not 
bother me. 

6. I liked seeing- how many words I could read 
in a minute or how many math problems I 
could finish in two minutes. 

7. I liked to see my graph of my reading or 
math scores. 

8. I would like to have timings again next year. 



SD 



2 
2 



SA 



2 3 4 
2 3 4 

2 3 4 



3 4 
3 4 



student Name 



Consultation - T 



We are interested in your perceptions of the instructional consultation project. 
Please circle the number on the ripht that best describes your agreement with 
each statement on the left. 



Note: SD « Strongly Disagree; D « Disagree; A = Agree; SA « Strongly Agree 



1. 



2. 



8. 



Curriculum-based measurement is a useful and 
helpful way to monitor student progress. 

Curri culm-based measurement is a useful way to 
determine student skill level. 

Curri culm-based measurement is useful in making 
instructional decisions. 

Computer printouts of observed student behaviors 
are useful for instructional planning. 

Instructional consultations are effective in 
developing student interventions. 

A collaborative consultation model is more 
effective than one in which the consultant is 
viewed as an expert. 

Sharing curri culu.n-based measurement 
graphs with students is a useful and motiva- 
tional teaching strategy. 

I would participate in a similar consultation 
program again. 



SD _D_ 
2 

2 

2 

2 

2 



A- SA 
3 4 

3 4 

3 

3 

3 



2 3 
2 3 

2 3 



9. What have been the most positive aspects of the instructional consultation 
program? 



10. What facets of the instructional consultation prograir. should be improved or 
changed? 



11. Any additional comments? 
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Social Validity Questionnaires for the Home-Based Consultation Intervention 



Homework - S: Student Questionnaire 
Homework - P: Parent Questionnaire 
Homework - T: Teacher Questionnaire 
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student Name 

Homework - S 

We want to know what you liked about the homework program: Please circle the 
number on the right that best describes your thoughts for each sentence. 



Note: oD - Strongly Disagree; D = Disagree; A « Agree; SA = Strongly Agree 



1. Since the beginning of the homework project 

a. I am given homework on a more regular basis. 

b. I am more willing to do my homework. 

c. I do more of my homework without help. 

d. I keep my homework materials better organized. 

e. I do more of my homework correctly. 

f. I get more of my assignment done each day. 

g. My parent (s) help me organize my homework more 
often. 

h. I enjoy doing school work more 



SD 



_D. 

2 

2 

2 

2 

2 

2 

2 
2 



jA_ 

7 

3 
3 
3 
3 
3 

3 
3 



SA 

4 

4 

4 

4 

4 

4 

4 
4 



2. I would like to be In a homework 
project like this one again. 



2 
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student Name 

Homework - P 

We are interested in your perceptions about the homework consultation program, 
Please circle the number on the right that best describes your agreement with 
statements on left. 



Note: SD « Strongly Disagree; D « Disagree; A - Agree; SA « Strongly Agree 



1. Since the beginning of the homework project ; 

a. The student is given homework on a more 
regular basis. 

b. The student is more compliant in getting 
started on his or her homework, 

c. The student has developed more independent 
study habits. 

d. The student's materials are better organized, 

e. The student completes a greater amount of his 
or her homework. 

f . The student is more accurate in his or her 
homework. 

g. The student's attitude is better toward 
homework. 

h. The student's attitude is better toward 
school • 

2. A consultant is helpful in planning homework. 

3. I would participate again in a similar 
homework intervention project 



SD JD_ 

2 

2 

2 
2 

2 

2 

2 

2 
2 



3 

3 

3 
3 

3 

3 

3 

3 
3 



SA 

4 

4 

4 
4 

4 

4 

4 

4 
4 



4. What have been ths most positive aspects of the homework consultation 

pr. gram? 



5. What facets of the program should be improved or changed? 



ERIC 



6. Any other comments? 
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student Name 



Homework - T 

We are interested in your perceptions of the homework consultation program. 
Please circle the number on the right that best describis your agreement with 
each statement on the left. 



Note: SD = Strongly Disagree; D = Disagree; A = Agree; SA = Strongly Agree 



1. Since the beginning of the homework project ; SD _D__ A SA 

a. The student is given homework on a more 

regular basis. 12 3 4 

b. The student is more compliant in getting 

started on his or her homework. 12 3 4 

c. The student has developed more independent 

study habits. 12 3 4 

d. The student' s materials are better organized. 12 3 4 

e. The student completes a greater amount of his 

or her homework. 12 3 4 

f . The student is more accurate in his or her 

homework. 12 3 4 

g. The student's attitude is better toward 

homework. 12 3 4 

h. The student's attitude is better toward 

school. 12 3 4 

2. A consultant is helpful in planning homework. 12 3 4 

3. I would participate again in a similar 

homework intervention project. 12 3 4 

4. What have been the most positive aspects of the homework consultation 
program? 



5. What facets of the program should be improved or chang e? 



6. Any other comments? 

17/JW.5.1 
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Appendix E 
Principars Social Validity Qucsilonnaircs 



I 



READING INTERVENTION 

Not Very 
at Al l Sometimes Frequently Frequently 

1. To what extent have you received 
feedback from teachers, students 

or parents regarding the project? 12 3 4 



2. To what extent have you been 

involved in the project? 1 2 3 4 



3. To what extent do you feel the 
project has benefited the 

student(s)? 12 3 4 



8. What have been the most positive aspects of the tutorial program? 



9. What facets of the program should be improved or changed? 



10. Any other comments? 
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INSTRUCTIONAL CONSULTATION 



1. To what extent have you received 
feedback from teachers, students 
or parents regarding the project? 



2. To what extent have you been 
Involved in the project? 



Not Very 

at All Sometimes Frequently Frequently 

12 3 4 

12 3 4 



3. To what extent do you feel the 
project has benefited the 

student(s)7 . 12 3 4 



8. What have been the most positive aspects of the consultation program? 



9. What facets of the program should be improved or changed? 



10. Any other comments? 
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HOMEWORK INTERVENTION 



Not Very 
at All Sometimes Frequently Frequently 

1. To what extent have you received 
feedback from teachers, students 

or parents regarding the project? 12 3 4 



2. To what extent have you been 

involved in the project? 12 3 4 



3. To what extent do you feel the 
project has benefited the 

student(s)? 12 3 



8. What have been the most positive aspects of the intervention? 



9. What facets of the intervention should be improved or changed? 



10. Any other comments? 
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